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Abstract 



In recent years there has been an increased interest in developing the basic 
reading and writing skills of primary students at-risk for academic failure. Problems 
associated with typical evaluations of programmatic impact on students’ literacy skills 
include the amount of testing time, sensitivity to smaller levels of growth, or, as in the 
case of the ISTEP+, the state-wide accountability program, the short time interval 
between the program intervention and the subsequent ISTEP+ testing. In this report we 
describe the use of the Basic Academic Skills Sample (BASS) in the evaluation of two 
literacy intervention programs. Results suggest that the BASS provides useful 
information about progress associated with early literacy interventions both for students 
as a whole and in comparing trends for students of differing ability levels. 



Using the Basic Academic Skills Samples (BASS) to Evaluate the Impact of 
Early Literacy Intervention Programs 



In recent years there has been an increased interest in developing the basic 
reading and writing skills of primary students at-risk for academic failure. It is generally 
assumed that providing a strong foundation in literacy early will better ensure more 
students will be successful in school and graduate from high school. With this in mind, in 
1997-98 the Indiana Department of Education implemented the Early Literacy 
Intervention Grant Program (ELIGP) to better meet the needs of the state’s early literacy 
challenge. Since 1997, funding has been provided by the state to develop early literacy 
programs. A substantial proportion of the funding went to the training of Reading 
Recovery (Clay, 1993) teachers. Schools had the option of choosing other locally or 
nationally developed early literacy interventions. Since then, implementation studies at 
the Indiana Education Policy Center (IEPC) have documented shifts in the nature of 
literacy instruction in the state (Manset, St. John, Simmons, Michael, Bardzell, Hodges, 
Jacob, & Gordon, 1999; St. John, Manset, Hu, Simmons, & Musoba, 2000). In addition, 
both the IDOE and schools have expressed interest in evaluating the impact of these 
curricular and organizational changes on student skill development. In response, we 
piloted the use of the Basic Academic Skills Samples (BASS) to evaluate their early 
literacy program. We were interested in finding an evaluation approach that would be 
both pragmatic and responsive to changes in programming. In this report, we summarize 
the first year of this pilot study. 

Evaluating Early Literacy Programs 

The proliferation of school reform models, debate over the most efficacious 
approach to reading instruction, and an increased emphasis on accountability all have led 
to an interest in evaluating early literacy programs. In Indiana, as in other states, students 
are now tested for the state assessment program at the beginning of Grade Three. The 
state assessment is too late to support a responsive change in programming, and in 
schools with a high attrition rate, the large percentage of the students in Grade Three that 
participate in the state assessment are not the same ones that received the literacy 
intervention. Some of the literacy interventions do include a form of formative 



evaluation, but while it is useful for tracking individual performance it is not always 
readily combined quantitatively to get a sense of the overall impact of programming 
changes. In addition, adding more standardized tests can be expensive and time- 
consuming for both teachers and students. In contrast to more formal, standardized 
assessment, curriculum-based assessment allows teachers to measure student progress in 
a way that is cost-effective, sensitive to performance changes, and can be modified to 
meet local needs (Howell, Fox, & Morehead, 1993). It also avoids the problems that arise 
with the use of high-stakes, standardized tests, particularly with ethnic minority students 
and students with disabilities (Manset & Washburn, 2000; Salvia & Ysseldyke, 1998). 

Researchers in special education recognized in the 1980s the need to develop a 
simplified way in which to evaluate program innovations for large numbers of students 
whose changes in performance are not easily registered by standardized tests. Deno 
(1985) and his colleagues developed the Basic Academic Skills Samples (BASS), a 
curriculum-based assessment tool that serves as an index for basic reading, writing, and 
math skills (Deno, Espin, & Maruyama, 1985). It is an index in that it selects a few 
examples of the skill that is then tested in a whole-class setting literally in minutes. 
Scoring is again relatively simple, yet the scores are predictive of those tests requiring 
more complex performance. While not appropriate for diagnostic decisions, the BASS 
has been used to evaluate progress for students with disabilities in full inclusion 
programs. For this study we were examining the use of the BASS as a means to evaluate 
the early literacy intervention programs of two schools. 

Method 

Participants 

A total of 618 students in two Indiana elementary schools participated in this 
study. Demographics and state achievement test scores can be found in Tables 1 and 2. 
Both are high poverty, urban schools that had received state grants to support the 
implementation of specific early literacy programs. Because of recent grant-supported 
changes in their early literacy instruction, both schools volunteered to participate in this 
pilot study examining the using of curriculum-based assessment (CBA) to evaluate their 
early literacy programs. In their grant application, Elm Creek Elementary School 
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described their reading instruction as a balanced combination of phonics, guided reading, 
independent reading, and process writing. They supplemented general reading instruction 
with Reading Recovery (Clay, 1993) and its extension, Literacy Groups, for the lowest 
readers. Reading Recovery is a highly-structured, intensive tutorial program designed to 
target Grade One students at-risk for reading failure. The program involves daily tutoring 
which includes the rereading of familiar books, the identification of letters and word 
making, the writing of a story, and performance assessment. Emphasis is placed on the 
development of student reading strategies. Literacy Groups were developed out of a need 
to extend the procedures of Reading Recovery to students in general and beyond Grade 
One. Elm Creek Elementary School proposed to use the grant money to increase the 
expertise of the paraprofessionals that were leading the Literacy Groups. All students in 
Grades One and Two at Elm Creek participated in the program. 

In their grant proposal. Sycamore Heights described their early literacy program 
as a combination of instruction from basal readers, trade books, and skill worksheets. 
They also stated there was no common philosophical or instructional approach shared by 
each teacher, which led them to choose a literacy program that would provide consistency 
across classrooms. Like Elm Creek, Sycamore Heights had Reading Recovery and 
Literacy Groups for Grade One students. They received grant monies to support the 
implementation of First Steps, a classroom-based language development model that 
targets reading, writing, spelling, and oral communication. First Steps is a complex 
program that includes intensive decoding instruction, Read Alouds, process writing, 
vocabulary development, and formative evaluation in whole class, small-group, and 
tutorial settings. The grant monies paid for professional development and materials to 
implement this program. In this first year of implementing First Steps, they chose to 
emphasize the writing portion of the program. Students in Grades One to Three were 
tested. 
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Table 1 

Demographics for Participating Schools 



School 


Grades 

Participating 


Enroll 


% Free 
Lunch 


% White 


% Black 


% Other 
Ethnic 


Elm Creek 


1-2 


674 


48.3 


87.5 


8.6 


3.8 


Sycamore 

Heights 


1-3 


569 


49.7 


82.8 


15.6 


1.6 



Table 2 

ISTEP+ Achievement Test Scores for Grade Three Students in Participating Schools 



School 


% Passing 
Language Arts 


% Passing 
Math 


Total 

NCE 


Elm Creek 


63 


70 


58.8 


Sycamore 


66 


86 


64.5 


Heights 









Instruments 

Students were tested in reading and writing using the BASS. The BASS is a 
curriculum-based assessment device designed to serve as an index of basic skill 
development. The reading portion consists of three maze passages with an approximate 
readability of Grade One. Students are allowed one minute to read and circle the correct 
multiple choice (CMC) provided for the blanks in each sentence. In the writing portion, 
students are given a story starter and have two minutes to write as much as they can of 
the story. Writing samples are given two scores, one for total words written (TWW) and 
total correct word sequence (CWS), a measure of the number of consecutive correctly 
spelled and grammatically correct word pairs. 

Procedures 

Students were tested in the beginning of November and May of the same school 
year. The first testing was administered by Policy Center staff while teachers observed. In 
the spring, classroom teachers administered the test. Policy Center staff scored the 
samples. Teachers also scored the spring sample, and their scores were compared with 
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those of the Center staff. Scoring by both teachers and Policy Center staff was 
comparable on the TWW and CMC, but there were unacceptable differences in the 
teacher and staff scoring of CWS. Because of the inconsistencies in the scoring of CWS 
by teachers, scoring by the Policy Center staff was used as fined scores in all cases. 

Study Design and Data Analysis 

Descriptive statistics were calculated for reading (CMC) and writing scores 
(TWW, CWS). Differences in fall and spring scores were statistically analyzed using 
dependent samples t-tests. Changes in scores were graphed by grade and by quartile 
within grade. Using a “lag” design, the statistical differences between Spring Grade One 
scores and Fall Grade Two scores, and Spring Grade Two and Fall Grade Three scores 
were calculated using independent samples t-tests as well. Although we acknowledge that 
this analysis is premature given the third data point that will be collected next fall, the 
findings were promising and therefore worth an examination. Because the interventions 
were designed to target the poorest readers, particular importance was paid to whether 
significant gains were made by students as a whole, whether the lowest-achieving 
students (Quartile 1) made greater gains than their higher-achieving classmates, and 
whether Quartile 1 students in Grade One exceeded the fall scores of Quartile 1 students 
in Grade Two. 



Results 

Elm Creek Elementary 

Descriptive statistics for Elm Creek’s reading and writing BASS scores can be 
found in Tables 3 and 4. In reading (CMC), mean gains for both Grade One (M=2.91, 
SD=3.83 to M= 11.27, SD=10.93) and Grade Two (M=9.37, SD=6.30 to M=23.99, SD= 
10.50) students were statistically significant (see Figure 1). Grade One students have 
already exceeded the November scores of Grade Two students. Significant gains were 
also made in Grades One and Two for writing: both Total Words Written and Correct 
Word Sequence (see Figures 2 & 3). 
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Table 3 

Descriptive Statistics for Elm Creek Reading and Writing Scores: Grade One 





CMC 


TWW 


CWS 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


All Students 














N 


112 


83 


112 


83 


112 


83 


M 


2.91 


11.27* 


12.57 


26.82* 


3.57 


12.26* 


SD 


3.83 


10.93 


8.26 


15.50 


4.00 


13.61 


QRTL 1 














N 


27 


27 


26 


19 


26 


26 


M 


0.00 


8.47 


5.85 


19.21 


0.00 


7.20 


SD 


0.00 


7.47 


2.63 


10.14 


0.00 


4.56 


QRTL 2 














N 


21 


21 


19 


19 


18 


18 


M 


1.00 


9.71 


10.11 


25.33 


1.06 


11.73 


SD 


0.00 


7.05 


0.81 


9.08 


0.24 


9.34 


QRTL 3 














N 


29 


29 


34 


34 


35 


35 


M 


2.57 


9.70 


13.56 


28.15 


3.17 


14.43 


SD 


0.63 


10.11 


1.65 


12.95 


0.98 


7.04 


QRTL 4 














N 


28 


28 


26 


26 


26 


26 


M 


8.32 


17.05 


24.12 


37.05 


8.65 


20.65 


SD 


3.96 


14.92 


7.58 


20.04 


3.70 


18.9 



*p<.01 (Analysis run for “All Students” only) 
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Table 4 

Descriptive Statistics for Elm Creek Reading and Writing Scores: Grade Two 





CMC 


TWW 


CWS 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


All Students 














N 


127 


127 


127 


127 


127 


127 


M 


9.37 


23.99* 


29.84 


52.00* 


14.73 


33.09* 


SD 


6.30 


10.50 


15.56 


18.79 


10.04 


15.09 


QRTL 1 














N 


24 


24 


28 


28 


27 


27 


M 


3.08 


14.33 


15.14 


43.65 


4.52 


20.00 


SD 


1.44 


8.48 


6.49 


14.51 


2.15 


10.09 


QRTL 2 














N 


29 


29 


32 


32 


31 


31 


M 


6.90 


25.94 


28.31 


44.05 


10.77 


34.04 


SD 


0.86 


11.06 


2.86 


12.92 


2.84 


15.76 


QRTL 3 














N 


35 


35 


29 


29 


30 


30 


M 


10.57 


23.20 


35.34 


60.58 


18.47 


31.78 


SD 


3.03 


5.98 


7.04 


22.43 


2.11 


11.26 


QRTL 4 














N 


30 


30 


29 


29 


30 


30 


M 


18.48 


29.78 


35.34 


60.36 


28.70 


42.90 


SD 


2.29 


11.37 


7.40 


17.31 


4.81 


15.17 



*p<.01 (Analysis run for “All Students” only) 
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Figure 1: Elm Creek Mean BASS Reading Scores, Grades One and Two 
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Figure 2: Elm Creek Mean BASS Total Words Written, Grades One and Two 
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Figure 3: Elm Creek Mean BASS Correct Word Sequences, Grades One & Two 
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Gains were also examined in reading and writing by quartile of performance on 
the fall scores. In Grade One, gains were made by students at all levels of performance 
(see Figures 4-6). The lowest-achieving students made similar gains to the higher- 
achieving students, and were able to achieve a mean (M =8.47, SD = 7.04) that was close 
to the average score of Grade Two students in the fall. In writing, for both measures of 
fluency (TWW) and mechanics (CWS), the lowest-achieving students made gains that 
were similar to those of their higher-achieving classmates. In both cases the lowest 
quartile students exceeded the mean scores for all Grade One students in the fall, and 
those of Quartile 2 students in the fall. 
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Mean Reading Scores 



Figure 4: Elm Creek Mean BASS Reading Scores, Grade One by Quartile 
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Figure 5: Elm Creek Mean BASS Total Words Written, Grade One by Quartile 
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Figure 6: Elm Creek Mean BASS Correct Word Sequences, Grade One by Quartile 
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For Grade Two at Elm Creek, significant gains were made in Reading (CMC) 
(M=9.37, SD = 6.30 to M=23.99, SD= 10.50), writing fluency (TWW) (M=29.84, SD = 
15.56 to M=52.00, SD=18.79), and writing mechanics (CWS) (M= 14.73, SD = 10.04 to 
M=33.09, SD=15.09) (see Figures 1-3). In reading, Quartile 1 students made gains 
similar to other groups of students, except the low-average, Quartile 2 students, which 
made exceptional gains that led them to exceed Quartile 3 students in the spring (see 
Figure 7). In Reading, Grade Two Quartile 1 students exceeded the average Grade Two 
students in the fall. Both fluency (TWW) and mechanics (CWS) writing gains of Quartile 
1 students exceeded not only the means of all students in the fall, but the TWW exceeded 
the scores of the highest achieving students (see Figures 8-9). 
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Mean Reading Scores 



Figure 7: Elm Creek Mean BASS Reading Scores, Grade Two by Quartile 




— QRTL 1 
— QRTL2 
— A— QRTL 3 
— X— QRTL 4 




18 



14 



Figure 8: Elm Creek Mean BASS Total Words Written, Grade Two by Quartile 
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Figure 9: Elm Creek Mean BASS Correct Word Sequences, Grade Two by Quartile 
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Sycamore Heights 

Descriptive statistics for Sycamore Heights reading and writing BASS scores can 
be found in Tables 5 -7. In reading (CMC), mean gains for both Grade One (M=1.92, 
SD=2.52 to M=8.74, SD=7.04), Grade Two (M=9.55, SD=7.47 to M=15.53, SD= 8.68) 
and Grade Three (M= 16.96, SD=7.51 to M=26.50, SD= 12.12) students were statistically 
significant (see Figure 10). Significant gains were also made in all three grades for 
writing, both Total Words Written and Correct Word Sequence (see Figures 1 1 & 12). 
These gains were particularly significant for Grade Three students. In Reading scores, 
mean Spring scores for Grade One students approached the Fall Grade Two students, and 
the mean Spring scores of Grade Two students approached the Fall scores of Grade Three 
students. In writing fluency (TWW), gains for Grade One and Grade Two students far 
exceeded those of the Fall scores of the students ahead of them. Gains in writing 
mechanics (CWS), were greater for Grade Three students than in either Grade One or 
Two. 

Grade One students in Quartile 1 made gains that were comparable to those of 
higher-achieving groups of students both in reading and writing scores (see Figures 13- 
15). These patterns are similar for Grade Two and Three students, with reading and 
writing gains for Quartile 1 students similar to higher-performing students. Quartile One 
students in Grades One and Two had higher mean scores than their Quartile One peers in 
the fall of the following grade level (see Figures 16-21). 

Discussion 

This pilot study involved the use of the BASS to evaluate the impact of changes 
in a school’s early literacy program. The findings represent the outcomes of data 
collected in early November and May. Students will be tested again next November. In 
this analysis, we are interested in whether the BASS can distinguish between grade levels 
of students, and ability levels within those grades. Mean scores were also examined for 
evidence of significant gains overall and for differently performing groups. Findings for 
reading, writing fluency and writing mechanics are discussed in light of the feasibility of 
using this method to evaluate early literacy reforms. 
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Table 5 

Descriptive Statistics for Sycamore Heights Reading and Writing Scores: Grade One 





CMC 


TWW 


CWS 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


All Students 














N 


136 


136 


136 


136 


136 


136 


M 


1.92 


8.74* 


11.87 


33.43* 


3.76 


14.41* 


SD 


2.52 


7.04 


7.60 


13.75 


3.19 


9.30 


QRTL 1 














N 


46 


46 


27 


27 


12 


12 


M 


0.00 


5.25 


3.33 


25.25 


0.00 


10.63 


SD 


0.00 


3.79 


1.41 


11.03 


0.00 


8.86 


QRTL 2 














N 


- 


- 


27 


27 


41 


41 


M 


- 


- 


3.33 


25.25 


1.49 


11.19 


SD 


- 


- 


1.41 


11.03 


0.51 


6.46 


QRTL 3 














N 


38 


38 


34 


34 


32 


32 


M 


1.53 


10.54 


12.06 


34.24 


3.84 


14.43 


SD 


0.51 


7.85 


1.84 


11.74 


0.85 


8.28 


QRTL 4 














N 


31 


31 


31 


31 


30 


30 


M 


5.26 


12.00 


22.32 


42.21 


8.27 


20.59 


SD 


2.54 


6.93 


4.48 


13.49 


2.15 


11.92 



*p<.01 (Analysis run for “All Students” only) 
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Table 6 

Descriptive Statistics for Sycamore Heights Reading and Writing Scores: Grade Two 





CMC 


TWW 


CWS 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


All Students 














N 


126 


126 


126 


126 


126 


126 


M 


9.55 


15.53* 


29.65 


44.42* 


23.88 


26.43* 


SD 


7.47 


8.68 


14.94 


17.79 


13.99 


16.83 


QRTL 1 
N 


25 


25 


26 


26 


28 


28 


M 


1.54 


9.25 


12.88 


30.57 


10.57 


13.96 


SD 


1.18 


6.36 


5.36 


11.20 


6.14 


9.67 


QRTL 2 
N 


22 


22 


29 


29 


29 


29 


M 


5.32 


11.15 


22.76 


40.12 


17.07 


22.50 


SD 


1.49 


5.75 


2.85 


14.11 


3.81 


12.66 


QRTL 3 
N 


37 


37 


30 


30 


24 


24 


M 


9.14 


14.06 


31.63 


47.54 


24.71 


27.00 


SD 


2.30 


5.47 


3.81 


16.60 


7.12 


14.87 


QRTL 4 
N 


29 


29 


28 


28 


32 


32 


M 


20.29 


25.48 


50.21 


57.37 


41.09 


39.74 


SD 


5.08 


6.90 


10.51 


14.98 


10.67 


15.82 



*p<.01 (Analysis run for “All Students” only) 
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Table 7 

Descriptive Statistics for Sycamore Heights Reading and Writing Scores: Grade Three 





CMC 


TWW 


CWS 




Fall 


Spring 


Fall 


Spring 


Fall 


Spring 


All Students 














N 


117 


117 


117 


117 


117 


117 


M 


16.96 


26.50* 


40.27 


52.91* 


26.32 


36.48* 


SD 


7.51 


12.12 


13.07 


16.36 


12.30 


16.70 


QRTL 1 














N 


24 


24 


25 


25 


25 


25 


M 


7.21 


17.25 


24.08 


41.52 


11.68 


19.73 


SD 


2.98 


12.37 


6.53 


12.16 


4.16 


8.45 


QRTL 2 














N 


26 


26 


27 


27 


27 


27 


M 


14.54 


24.13 


35.67 


48.04 


21.26 


34.04 


SD 


1.42 


6.60 


2.06 


12.72 


2.52 


10.16 


QRTL 3 














N 


15 


15 


26 


26 


27 


27 


M 


17.60 


25.50 


43.69 


56.91 


29.78 


43.63 


SD 


0.51 


5.92 


1.83 


12.81 


2.06 


12.01 


QRTL 4 














N 


33 


33 


27 


27 


26 


26 


M 


25.30 


35.38 


56.56 


66.24 


42.08 


50.48 


SD 


4.97 


11.79 


8.67 


16.47 


9.43 


17.25 



*p<.01 (Analysis run for “All Students” only) 
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Figure 10: Sycamore Heights Mean BASS Reading Scores, Grades One to Three 
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Figure 11: Sycamore Heights Mean BASS Total Words Written, Grades One to Three 
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Figure 12: Sycamore Heights Mean BASS Correct Word Sequences, Grades One to Three 
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Mean Reading Scores 



Figure 13: Sycamore Heights Mean BASS Reading Scores, Grade One by Quartile 
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Figure 14: Sycamore Heights Mean BASS Total Words Written, Grade One by Quartile 
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Figure 15: Sycamore Heights Mean BASS Correct Word Sequences, Grade One by Quartile 
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Figure 16: Sycamore Heights Mean BASS Reading Scores, Grade Two by Quartile 
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Figure 17: Sycamore Heights Mean BASS Total Words Written, Grade Two by Quartile 
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Mean Correct Word Sequences 



Figure 18: Sycamore Heights Mean BASS Correct Word Sequences, Grade Two by Quartile 
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Figure 19: Sycamore Heights Mean BASS Reading Scores, Grade Three by Quartile 
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Figure 20: Sycamore Heights Mean BASS Total Words Written, Grade Three by Quartile 




Q1 

-rn-Q2 

-A-Q3 

-X-Q4 




J J 



31 



Mean Correct Word Sequences 



Figure 21: Sycamore Heights Mean BASS Correct Word Sequences, Grade Three by Quartile 
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Administration of the BASS was fairly simple and involved minimum teacher or 
student time. Instructions were relatively simple and protocols easy to follow. Teachers 
and Policy Center staff reported that the lowest-performing students in Grade One were 
frustrated in that they could not read any words in the passages. In Sycamore Heights 
more than a third of the Grade One students were also reported to be frustrated with being 
cut short and not completing the reading or writing passage. Given, however, the 
necessity of using a similar task for each grade level in order to make the cross-grade 
comparison, there may not be an alternative to the “power” (i.e., timed) aspect of the test. 

Calculating reading and writing fluency scores was fairly simple, and teachers and 
Policy Center staff were able to reach consistent, reliable scores with only approximately 
20 minutes of training. The measures of writing mechanics, CWS, was relatively 
complicated to calculate and it was difficult to reach an appropriate level of reliability 
between teachers and research staff. Particularly when there is limited opportunity to 
train and practice scoring, it would be more advisable to simply calculate correctly 
spelled words. While this does not allow for a documentation of progress in aspects of 
writing mechanics other than spelling, at this skill level it will be easier to obtain 
consistency in scoring. 

It is clear that significant gains were made in reading, writing, fluency and 
mechanics in both schools. In Elm Creek, Grade One students already exceeded the Fall 
reading scores of Grade Two students. While it is not possible to credit directly changes 
that occurred due to the ELIGP grant supported literacy program, there is some evidence 
that changes in the programs as a whole were sufficient to support the reading 
performance of Grade One students. In writing fluency and mechanics, scores for Grade 
One students are approaching the Fall scores of Grade Two students. At this rate they 
have the potential to outperform the older cohort of students in writing as well. 

At Sycamore Heights, the reading scores of both Grade One and Two students 
approached those of the Fall scores of their relatively older cohort of students, suggesting 
that changes in literacy instruction may be having a positive effect. The greatest gains 
were made in Grade Three, but because there is no reference it is difficult to rule out that 
as being a function of grade level. There is evidence of more impact on students’ writing 
fluency, where scores of students in Grades One and Two are already exceeding the Fall 
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scores of students in Grades Two and Three, respectively. While again, conclusions 
should be drawn cautiously, it is interesting to note that at Elm Creek, where there was an 
effort to focus on reading, there are indications of a positive impact on Grade One 
reading skills. On the other hand, at Sycamore Heights, where the focus this year was on 
writing instruction, writing fluency has increased. 

In both schools, students at all levels of ability made gains. This suggests that 
instruction has positively affected almost all students, no matter their ability. However, 
for the lowest-achieving students, if they start out behind their peers the gap between 
high and low readers will be compounded over the years. An ideal pattern would be gains 
for all ability levels; but relatively greater or accelerated gains for the lowest-achieving 
students. 
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